Aiming at the problems that traditional interference alignment algorithm based on the maximum Signal to Interference and Noise Ratio (SINR) in Multiple-Input Multiple-Output (MIMO) cognitive network is hard to converge when sending multiple data streams and the interference between them is prominent, an interference alignment algorithm that considers data stream interference and iterative limit was proposed. Firstly, the secondary users eliminated interference between primary users and secondary users through coding design. Then, when eliminating the interference between the primary users and the secondary users, the Generalized Rayleigh Entropy (GRE) was used to calculate the precoding and interference suppression matrix based on the maximum SINR algorithm according to channel reciprocity, and in the iterative process, each iteration always made precoding and interference suppression matrix firstly satisfy that the interference power in the expected signal space was minimal. Finally, combined with the MIMO interference channel between the secondary users, the interference channel between primary and secondary users and the necessity of interference alignment of secondary usernetwork, the secondary users' reachable upper bound of degree of freedom was deduced. The experimental results show that compared with the traditional maximum SINR algorithm, the proposed algorithm has no significant improvement in the total capacity of the secondary users when the signal to noise ratio is low, but with the increase of signal to noise ratio, the advantages of the proposed algorithm are more and more obvious. When convergence is reached, the iterative times of the proposed algorithm are reduced by 40% compared with the conventional maximum SINR algorithm. Therefore, the proposed algorithm can improve system capacity and accelerate convergence.

Concerning the demands of large capacity and high spectrum utilization in future mobile communication system, a method for optimizing the capacity of secondary user system in Multiple-Input Multiple-Output (MIMO) cognitive networks based on Non-Orthogonal Multiple Access (NOMA) was proposed. Firstly, the transmitted signals were pre-coded, and then the cognitive users were clustered according to channel gains. Secondly, the power allocation was performed for users after clustering. Finally, the Non-deterministic Polynomial-hard (NP-hard) multi-cluster objective function was transformed into solving the capacity of each sub-cluster. Meanwhile, taking into account Quality of Service (QoS) of cognitive users and requirement of Successive Interference Cancellation (SIC), the optimal power allocation coefficient, which is a constant between 0 and 1, was solved by using Lagrange function and Karush-Kuhn-Tucker (KKT) condition. The simulation results show that, the proposed method outperforms the average power allocation method. And when the channel quality is poor, compared with the MIMO cognitive network based on Orthogonal Multiple Access (OMA), the proposed method has improved the capacity of secondary user system significantly.